Ultrasound fusion imaging method and ultrasound fusion imaging navigation system

ABSTRACT

The present application relates to an ultrasound fusion imaging method and an ultrasound fusion image navigation system. The ultrasound fusion imaging method includes a selection step, a registration step and a fusion step. The selection step is for selecting at least one ultrasound image from at least one previously stored piece of ultrasound video data according to an input instruction. The ultrasound video data includes an ultrasound image obtained by acquiring a target object in at least one plane and position indicating information corresponding to the ultrasound image. The registration step is for registering the selected at least one ultrasound image with a modality image. The registration process uses the location of the position indicating information of the at least one ultrasound image. The fusion step is for fusing the registered ultrasound image with the modality image.

TECHNICAL FIELD

The present disclosure relates to medical ultrasound imaging, inparticular to fusion methods for fusing ultrasound images with apre-stored modality image and an ultrasound fusion imaging navigationsystem.

BACKGROUND

More than one kind of image-generating system for sampling target objectis able to be implemented in clinical, so that doctor can acquiremultiple models of medical images, such as Computed Tomography (CT),Magnetic Resonance (MR) or ultrasound image. The principle of ultrasoundimage fusion navigation is, through space positioning devices (usuallyMagnetic locating sensor fixed on a probe), to built a spatialcorresponding relation to real-time ultrasound images with otherpre-acquired modality data (such as CT or MR image) and display theoverlapped cutting sections images with the pre-acquired modality datafor achieving the fusion of above two kinds of images. Under above,these two kinds of images are used both in diagnosis process andtreatment process to combine the high resolutions feature of CT or MRwith the real-time feature of ultrasound image so as to provide moredetail diagnosis information for doctor in clinical to increase theeffect of treatment.

In the ultrasound image fusion navigation system, the most importanttechnique point is to register ultrasound images with the modality data.In detail, the practice of above registering point is to map theposition of the point (or plane) of the ultrasound images in a worldcoordinate system with the position of the point (or plane) of themodality image in a world coordinate system correspondingly. Acquiringtarget position in the world coordinate system precisely is insignificant effect for increasing the accuracy of registration.

The conventional registration technique is based on real-time ultrasounddetection; doctor acquires ultrasound images for providing registeringinformation by freezing the present frame. This kind of method,processing the real-time ultrasound image frame by frame, is well knownfor those skilled in the art. In addition, if doctor wants to acquirecertain section image in certain breath depth, patient usually is askedto cooperate by precisely controlling his breath. Especially, when imagefusion of abdominal organs for those patients, who's breath way is inabdominal breath way, is conducted, huge errors is generated because ofthe movement, rotation and deformation caused by the abdominal breath ofpatient. Therefore, patient is asked to precisely controlling hisbreath. It raises the requirements for both doctor and patient. Ifpatient cannot control his breath well enough or doctor has a lack ofexperience, the effect is usually not satisfied so as to reduce theaccuracy of registration and the successes rate for image fusion. Rightnow, the method for eliminating breath effect is depending on doctor'sdecisions by manually determining breath phase or adding a sample sensorfor sample breath control. However, the performances of above solutionsare all poor and unsatisfied.

SUMMARY

Therefore, an ultrasound fusion imaging method and an ultrasound imagingnavigation system for eliminating or reducing breath effect areprovided.

A method for fusing at least one ultrasound image and a pre-storedmodality image comprises a selection step, a registration step and afusion step. The selection step is for selecting at least one frame ofultrasound image from at least one pre-stored ultrasound video dataaccording to an input instruction. The ultrasound video data comprisesultrasound images acquired by sampling a target object from at least oneplane and position-indicating information corresponding to each of theultrasound images. The position-indicating information is generated by aposition sensor, fixed with an ultrasonic probe, according to a motionstate of the ultrasound probe sensing by the position sensor during theacquisition of the ultrasound images. The registration step is forregistering the selected at least one ultrasound image with the modalityimage. The at least one position-indicating information is implementedin above registering process. The fusion step is for fusing theregistered at least one ultrasound image with the modality image.

In one embodiment of the method for fusing at least one ultrasound imageand pre-acquired modality image of the present invention, multipleframes of ultrasound images are selected in the selecting step. Themethod further comprises a breath model built step and abreath-correcting step. The breath model built step is for building abreath model according to the ultrasound video data. Thebreath-correcting step is conducted before the registering step orduring the fusion step for implementing the breath model to correct themultiple frames of ultrasound images into the same breath depth level.

A method for fusing at least one ultrasound image with a pre-storedmodality image including a selection step, a breath model built-up step,a registering step and a fusion step is provided.

The selection step is for selecting multiple frames of ultrasound imagesfrom at least one portion of ultrasound video data. The ultrasound videodata comprises ultrasound images acquired by sampling the target objectfrom at least one plane and position-indicating informationcorresponding to each frame of the ultrasound images. Theposition-indicating information is generated by a position sensor fixedwith an ultrasonic probe while acquiring the ultrasound images.

The breath model built-up step is for building a breath model accordingto the ultrasound video data;

The registering step is for registering the multiple frames ofultrasound images with the modality image;

The fusion step is for fusing the registered multiple frames with themodality image. The breath model is implemented to correct the multipleframes of the ultrasound images into the same breath depth before theregistering step or in the fusion step.

An ultrasound fusion imaging navigation system, comprises: an ultrasoundprobe and a position sensor fixed with the probe; a sampling moduleimplemented for sampling the target object from at least one plane togenerate at least one ultrasound video data comprising registeringinformation and storing position-indicating information for each frameof ultrasound images in each ultrasound video data, wherein positionindicating information is generated by the position sensor according tomotion of the ultrasound probe in the acquiring process of theultrasound images; a replaying module for replaying the pre-storedultrasound video data according to an input instruction; a selectingmodule for selecting at least one frame of the ultrasound images fromthe replaying ultrasound video data according to a selectioninstruction; a registering module for registering the selected at leastone frame of ultrasound images with a modality image, wherein theposition-indicating information of the at least one ultrasound images isimplemented in above registering process; and a fusion module for imagefusing the registered ultrasound images with the modality image.

A distinct image fusion method comparing with conventional real-timebase fusion method is implemented by the ultrasound fusion imagingmethod and an ultrasound imaging navigation system of the presentinvention. The fusion method disclosure in the present inventioncomprises a step of sampling the target subject and pre-recording avideo data accordingly before the registering process, and selecting oneor multiple ultrasound images thereafter. Therefore, the performance forreducing the target object breath effect in the embodiment of thepresent invention is significantly increased.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram of an ultrasound fusion imagingnavigation system in accordance with an embodiment of the presentdisclosure;

FIG. 2 is a schematic flow chart of a fusion method for fusingultrasound images with a pre-stored modality image in accordance with anembodiment of the present disclosure;

FIG. 3 is a schematic block diagram of spatial switching in accordancewith an embodiment of the present disclosure;

FIG. 4 is a schematic block diagram for displaying multiple registeringplanes in accordance with an embodiment of the present disclosure;

FIG. 5 is a schematic flow chart of a building breath module inaccordance with an embodiment of the present disclosure;

FIG. 6 is a schematic block diagram of a breath depth relating to timeaxis according to an embodiment of the present disclosure;

FIG. 7 is a schematic block diagram of the displacement of a targetobject relating to a reference position through linear fitting withrespect to 3 dimensions of the world coordinate system corresponding tochange of the breath depth according to an embodiment of the presentdisclosure;

FIG. 8 is a schematic flow chart of a registering and fusing processafter the breath module is built according to an embodiment of thepresent disclosure;

DETAILED DESCRIPTION

Specific details for fully understanding each of embodiments andimplemented by those skilled in the art are provided in the belowdescription. However, it should be understood for those skilled in theart that the present invention is able to be implemented without thespecific details as well. In some embodiments, conventional structuresand functions are omitted to avoid confusions in the descriptions of theembodiments.

Unless it is acquired clearly under context of the descriptions, theterms “comprise”, “include” should be defined as opening definition butnot limited or exhaustive definition.

A schematic block diagram of ultrasound image fusion navigation systemis shown in FIG. 1. An ultrasound probe 101 emits ultrasound to at leastone target organ of human body. An ultrasound imaging module 102 isimplemented for received echo wave signal of the ultrasound emittingfrom the ultrasound probe 101 to generate at least one ultrasound imageof the target organ. At least one pre-stored modality image, such as CTimage or MR image, is imported into a registering fusion module 105. Aposition sensor 103 fixed with the ultrasound probe 101 senses positionof the probe 101 and provides position information constantly along withmotion of the ultrasound probe 101. The position information comprisesinformation relating to 6-axises space position information of theultrasound probe 101 (including but not limiting to vertical direction,horizontal direction, portrait direction, pitch direction, rolldirection and sway direction-). The registering and fusion module 105registers and fuses the ultrasound image with a modality image byimplementing the image information and the position information.

Combining with the ultrasound image fusion navigation system shown inFIG. 1, a schematic flow chart of a fusion method of the ultrasoundimage and pre-stored modality image is provided and shown as FIG. 2,comprising step S11 to S13 as disclosed below: a selection step S11,selecting at least one frame of ultrasound image from a pre-storedultrasound video data according to an input instruction.

The pre-stored ultrasound video data comprises pre-sampling data oftarget object (target organ, such as liver) so as to generate built-inregistering information (registering video). Position indicatinginformation R_(probe)(t), which relates to position of the positionsensor 103 fixing with the ultrasound probe 101, is recordedsynchronically for each frame of ultrasound images t. Theposition-indicating information R_(probe)(t) is generated by theposition sensor 103 fixing with the ultrasound probe 101 according tothe motion state of the ultrasound probe 101 in the process of acquiringthe ultrasound images. It means, the content in the registering videodata comprises at least both the ultrasound images data and theposition-indicating information R_(probe)(t) of the position sensor 103in addition. In one embodiment, the position sensor 103 is capable ofapplying as an electromagnetic induction base position sensor, or anoptical base position sensor as well. An acoustics base position sensoris also able to be applied into this embodiment. For explaining thepresent invention but not as limitations, electromagnetic induction baseposition sensor is implemented for describing in below embodiments ofthis application.

In the selection step, the input instruction is capable of selectingfrom external user input, or automatically triggering the instructioninside the system when the registering and fusing is processed. In oneembodiment, the pre-stored ultrasound video data is replayed, and theselection step is conducted thereafter in above replaying process.Commands for playing and selecting the ultrasound video data aredetermined under user's demands. Embodiments of how to play and selectthe ultrasound video data are able to be implemented under conventionaltechnologies, just only needs to satisfy functions suitable forreplaying ultrasound video data and selecting image frames. In aboveultrasound replaying step, the pre-stored ultrasound video data could bereplayed one frame by another, from beginning to the end, or could bedirectly dragged to a specific frame containing valuable target organcutting-section image via program tool bar or mechanical switch button.In another embodiment, the ultrasound video data is also selected underpre-determined conditions, for example, some specific frames such as 20frames ahead in the ultrasound video data are capable of beingpredetermined to be selected.

In the registering step S12, at least one selected ultrasound imageframe is registered with the modality image wherein the positionindicating information of the corresponding ultrasound image frames areimplemented in this step in the mean time.

In various embodiments, the three dimensional ultrasound imaging systemmay scan the head of a fetus, i.e., may transmit ultrasound wavestowards the head of the fetus and receive ultrasound echoes, to obtainultrasound echo signals. The ultrasound echo signals may be processed asdescribed above and thereby three dimensional amount data of the head ofthe fetus (hereinafter, “three dimensional amount data”) may beobtained. The specific processes for scanning the target and processingthe ultrasound echo signals to obtain the three dimensional amount datacan be the same to or similar with those which are well known in the artand thus will not be described in details herein.

For the purposes of increasing total accuracy of the system, two or moreultrasound image frames in the modality image are selected in theregistering step S12 in another embodiment. It should be noticed thatmultiple frames of ultrasound images can only be applied in theregistering step S12 only on the condition that above ultrasound imagesframes are sampled under the same or similar breath state (depth).Generally, if ultrasound images frames are all sampled under the statewhen patient temporally stops his breath, each ultrasound image frame ofthe video data should be considered under similar breath state so ascould be directly applied to register at the same time in theregistering step S12.

In the registering step S12, a spatial transformation algorithm isimplemented for mapping one image to a frame image so that each pixel ofabove two images corresponding to the same position in one spacecoordinate system could be linked correspondingly. Therefore, aboveimage information could be fused with corresponding images correctly.Under above described algorithm, in the ultrasound image fusionnavigation system of the present invention, the registration process forthe ultrasound image and the modality image could be implemented via aspatial transformation reference shown in FIG. 3. In the other word,each pixel of the ultrasound images is transformed from the coordinatesystem of its own to the coordinate system of the position sensor 103 atfirst, and then transformed from the coordinate system to WorldCoordinate System (WCS) thereafter. In the end, each pixel of theultrasound images is transformed from WCS to the coordinate system ofthe modality image. In this and other embodiments, WCS means a referencecoordinate system, which is definable under requirements, such as acoordinate system of a magnetic field generator or other coordinatesystem is also applicable for sure. The spatial transformation referenceshown in FIG. 3 could be described as a mathematical formula formation:X _(sec) =P·R _(probe) ·A·X _(us)  (1)

Wherein, X_(us) is the coordinate of one position in the ultrasoundimage, X_(sec) is the coordinate indicating to the same position in themodality image, A is a transformation vectors matrix from the ultrasoundimage space coordinate system to the position sensor space coordinatesystem, R_(probe) is a transformation vectors matrix from the spacecoordinate system of the position sensor to the space coordinate systemof the magnetic field generator, P is a transformation vectors matrixfrom the reference coordinate system to the space coordinate system ofthe modality image.

Regarding to the configuration of the transformation vectors matrix A,since the position sensor 103 is fixed with the ultrasound probe 101,when the ultrasound probe 101 is hold in a constant depth, thetransformation vectors matrix A is fixed as well. Therefore, thetransformation vectors matrix A is acquirable through calibrationalgorithm by combining the position indicating information R_(probe)(t)before registering.

Regarding to the configuration of R_(probe), it could be directlyaccessed from a locating controller 104 electrically connected with theposition sensor. In this embodiment, R_(probe) is varying constantlywith the motion of the ultrasound probe 101.

Regarding to the configuration of P, which is also defined asregistration matrix, could be acquired by applying formula (1) viafinding corresponding point or plane between the coordinate spaces ofthe ultrasound image and the modality image. In one embodiment, somespecific point or area of the target organ are marked. Above markedpoint or area of the target organ are sampled to generate at least onepoint or area X_(us) in the coordinate space of the ultrasound image andat least one point or area X_(sec) in the coordinate space of themodality image so as to generate P from above through applying Formula(1). In another embodiment, some points or areas of the coordinate spaceof the ultrasound image are transformed to WCS, and some points or areasof the coordinate space of the modality image are also transformed toWCS thereafter so that X_(sec) in the coordinate space of the modalityimage, corresponding to X_(us) in the coordinate space of the ultrasoundimage, is acquired through image mapping algorithm. Thereafter, P isgenerated from X_(sec) and X_(us). For those skilled in the art of thisfield, it is easy to reach that a reversing transformation algorithmcould be applied to acquire a reversing coordinate transformation fromone coordinate space of image to another.

In step S13, image fusion is conducted between the registered ultrasoundimage and the modality image. Above image fusion step could beimplemented by referring to conventional image fusion algorithm, such asimage fusion based on empty space algorithm, maximum (minimum)gray-scale value algorithm, weighted gray-scale value algorithm, orimage fusion based on domain transformation, such as multi-resolutionpyramid algorithm or Fourier transformation.

Under above steps, the ultrasound images and the modality image areregistered and fused. The implemented image fusion algorithm here isdifferent from conventional applications. In detailed, conventionalapplication is based on real-time ultrasound process of dealing(freezing) frames one by one. On the other hand, the embodiment isimplemented by recording (pre-stored) a video data before theregistering S12 step then selecting one or more frames of the video datafor registering via replaying the video data.

Except above steps, the image fusion method for fusing the ultrasoundimage and the modality image further comprises below steps: amulti-frames displaying step, displaying intersection lines and includedangles among these frames at the time when the multi-frames areregistered or fused. As shown in FIG. 4, an intersection lines position305 among different frames of the ultrasound images and an includedangle 307 between two frames are illustrated in the right block and theleft block individually. If displaying of multiple registered or fusedframes is needed, it could be implemented by selecting one of themultiple frames as a reference frame and defined its intersection linesand the included angle as intersection lines and included angle amongeach of the remaining frames and the reference frame. One of theadvantages in this embodiment is relative positions among the frames areindicated more intuitively so as to increase performance for theregistering step S12 thereafter.

The breath-controlling ability of many patients is usually lower thanits normal state when the ultrasound image fusion navigation system isimplemented. On the other hand, the pre-stored video data forregistering is sampled and fused when the breath-controlling ability ofthe patients is under normal and free condition. Therefore, the effectcaused under different breath conditions of patients in registering andfusing processes are significant and should be reduced or eliminated,because complex motions are caused among organs of patients such asmovement or rotation defined as rigid motion or entirely deformation, orpartial deformation caused by extrusion among patients organs, definedas non-rigid motions. Base on above issues, a method for registering andfusing the ultrasound image with the pre-stored modality image isprovided in this embodiment. A breath state model for describing themotion of organs with patient breath is built based on the positionindicating information of the position sensor and the ultrasound videodata at first. After that, a time-varying correction spatial modelacquired through the breath state model is implemented for registeringand fusing to reach the purpose for reducing and eliminating ofbreath-effect.

The method for reducing and eliminating breath effect implemented inthis embodiment is described in mathematical formula as:X _(sec) ·P·T(R _(probe) ·A·X _(us))  (2)T is a kind of spatial mapping algorithm for correcting, selected formlinear mapping, affine mapping and other non-linear mapping algorithms.In general, T is defined as a random continuous mapping among algorithmthree-dimensions. Specifically, T can be a space mapping matrix forcorrection.

Generally, breath motion is more regular under free-breathing conditionand is able to be defined as a periodic motion approximately. Whenpatient breaths, his abdominal skin mainly moves along with back-forwarddirection and is defined as a back-forward reciprocating motionapproximately. For those target organs that its motions are mainlycaused by breath, similar to abdominal skin motions, and its motions arecapable of defined as a periodic motion as well. A linear model isimplemented to describe motions of this kind of target organs versusbreath motion under the condition that motions of this kind of targetorgans are rigid motions. Linear model is imported for explaining inthis embodiment. In those embodiments that motions of this kind oftarget organs contains non-rigid component, conventional non-rigidalgorithm is combined to resolve non-rigid component issue. Regarding tolinear model, each point in space shares the same mapping relationship,shown as Formula (3):X _(sec) ·P·T·R _(probe) ·A·X _(us)  (3)Wherein the spatial mapping T degenerates as a matrix.

For the purpose of building the breath model, a breath sensor 109 isimplemented into the embodiment shown in FIG. 1. the breath sensor 109is fixed on the patient's abdominal skin for following the motions of aposition the breath sensor 109 fixed on so as to sample how aboveposition moves with patient breath.

For the understanding of following descriptions, relative terminologiesare defined below:

-   (1) Reference position: any position on the motion path of the    breath sensor moving with abdominal skin, such as a middle position    on the motion path of the breath sensor;-   (2) Breath depth: the movement of the breath sensor relative to    reference position, implemented for describing the state of breath    approximately, defined as d(t). Breath depth is acquirable by    referring to the breath position information R_(resp)(t) and    implemented through conventional transformation method to transform    position information of the breath sensor.-   (3) Reference breath depth: reference breath depth is the breath    depth relates to the breath depth corresponding to the reference    position, defined as d₀-   (4) Relative motion amount of reference coordinate system: the    position of the target organ under the reference breath depth d₀ is    defined as the reference position. The motion amount relates to the    reference position under different breath depth d(t) in the world    coordinate system is defined as relative motion amount of reference    coordinate system. For explaining, rigid motion is applied as    example in this embodiment and rotation motion is ignored but only    relative motions of the reference coordinate system are considered.    The relative motion amount of reference coordinate system is defined    as W(d(t)) in this embodiment.

As illustrated in FIG. 5, the method for building the breath model inthis embodiment comprises below steps S21˜S25. Step 21, configuring thereference breath depth and the reference position in accordance with thepre-stored ultrasound video data. Pre-stored ultrasound video data isgenerated by sampling data for patient abdomen from one plane ormultiple planes (such as n planes, n is a positive integer) overlappingeach other with included angles, wherein sections of ultrasound videodata are sampled from each planes separately. That means one portion ofthe ultrasound video is sampled from one corresponding planeindividually so as to generate n sections of ultrasound video dataUSV_(i), i=1, . . . , n, frame numbers of the ultrasound video datapresents by variable t. Variable USV_(i)(t) indicates the t frameultrasound image in the i portion of the ultrasound video data. In themeantime, the breath position information R_(resp)(t) and the positionindicating information R_(probe)(t) in each frame of sampled ultrasoundimages are recorded. In each sampling process for each ultrasound video,sampling period contains one or more period of breath.

As described above, the feature of breath motion is its periodicity.This periodic characteristic is similar with the sine wave shown in FIG.6. The horizontal axis t indicates breath period, and the vertical axisindicates breath depth d(t). Each repeating curve indicates a breathperiodic. As shown in FIG. 6, transverse broken line is defined as thereference position do, and breath depth varies with frame numbers t ofvideo data. The reference position is defined as the middle position inthe motion path of the sensor in this embodiment. Apparently, thereference position could be defined as the lowest or the highestposition in the motion path of the sensor as well. Since the same breathdepth might relates to inhaling state or exhaling state, the breathstate could be further distinguishes as inhaling phase and exhalingphase so as to generate sine wave curve similarly.

In step S22, a reference frame is selected corresponding to each portionof video data to acquire motion amount Vi(d(t)) relating to thereference frame for a target object corresponding to other frames ofultrasound image.

For each portion of ultrasound video data USV_(i), if one frame of theultrasound image corresponding to breath depth d₀ is selected as thereference frame, the motion amount V_(i)(d(t)) relating to the referenceframe for a target object corresponding to other frames of ultrasoundimage is acquired through motion tracing via conventional algorithm suchas model mapping algorithm.

Step S23 comprises step that transforming image data into the samereference coordinate system to reduce or eliminate the quivering effectof the ultrasound probe. Since the ultrasound probe is hard to guaranteeit's still state for sure when the breath correcting video data issampled, the motion following process should be conducted only restrictin the plane of the reference frame to eliminate the quivering effect ofthe ultrasound probe. In step S23, if x₀ is the position of one point ofthe reference frame and x(t) is the mapping position of above positionof one point acquired by the motion following process in the t frame ofultrasound image. If R_(resp)(t) is defined as the breath positioninformation and R_(probe)(t) is defined as the position indicatinginformation, the corresponding points of x₀ and x(t) are defined as m₀and m(t), then:m ₀ =R _(probe_0) ·A·x ₀  (4)m(t)=R _(probe)(t)·A·x(t)  (5)

if the projection component of W(d(t)) of other frames (not thereference frame) in the plane corresponding to the reference frame isdefined as proj_(i)(W(d(t))), V_(i)(d(t)) is the observed value of theprojection component of the projection component proj_(i)(W(d(t))).Under above configuration, the projection component of m(t)−m₀ in theplane corresponding to the reference frame is proj_(i)(m(t)−m₀). It isthe common mathematical function of the observed value V_(i)(d(t)) ofproj_(i)(W(d(t))):V _(i)(d(t))=proj_(i)(m(t)−m ₀)  (6)

In this embodiment, the reference frame and those non-reference framesare both transformed into the same world coordinate system forprojecting process so as to eliminate the shift error caused fromquivering probe. Device such as probe clip is implemented to fix theposition of probe so as to hold the position of the probe as possible.If the patient stays still, the position of the probe could be definedas a constant in sampling process, and R_(probe)(t)=R_(probe_0), andV_(i)(d(t))=x(t)−x₀.

In step S24, in accordance with V_(i)(d(t)) acquired in step S22, therelative motion W(d(t)) of the target organ corresponding to otherframes of ultrasound image in different breath depth is calculated underthe same reference coordinate system.

When all sections of video data are configured at the same referencebreath depth, presenting as d(t)=D, the relative displacement of thetarget organ corresponding to the reference coordinate system of thereference position is defined as Formula (7) shown below throughoptimization, wherein W(D) is the corresponding value when the outmostlayer Σ is configured as its small value:

$\begin{matrix}{{W(D)} = {\arg\;{\min\left( {\sum\limits_{i = 1}^{n}\left( {\sum\limits_{{{All}\mspace{11mu}{d{(t)}}} = D}{{{{proj}_{i}\left( {W(D)} \right)} - {V_{i}\left( {d(t)} \right)}}}^{2}} \right)} \right)}}} & (7)\end{matrix}$Wherein arg min( ) is defined as the function acquiring the smallestvalue for function in its brackets. Formula (7) is resolved to getdisplacement W(d) under multiple breath depth d(t).

In step S25, step fitting different breath depths and its correspondingreference coordinate systems is conducted to acquire a breath model.

Above “breath model” is defined as discipline for displacement of thetarget organ varying with the breath depth. The term “building breathmodel” is defined as to acquire the mathematical expression of thediscipline for displacement of the target organ varying with the breathdepth in accordance with the pre-stored ultrasound video data orobservation for above discipline.

In this step, point (d, W(d))) is fitted in certain way, wherein d isdefined as a self-varying value, so that the discipline for displacementof the target organ varying with the breath depth is acquiredcorrespondingly to get the mathematical expression of it.

Differential mode of two vectors is implemented to determined error∥proj_(i)(W(D))−V_(i)(d(t))∥² between the projections proj_(i)(W(D)) ofV_(i)(d(t)) and W(d(t)) in the plane i under certain breath depth D inthis embodiment. The margin of the error is also could be described indifferent way, such as ∥proj_(i)(W(D))−V_(i)(d(t))∥, in otherembodiments.

The schematic diagram of the three straight lines shown in FIG. 7 is thediscipline of target organ corresponding to the reference position inthree dimensions of world coordinate system with breath depth d. Otherfitting methods could be implemented in other embodiments. For example,other curves such as conic curve or cubic curve are also able to beimplemented for describing.

In this embodiment, the breath model is implemented for correcting afterthe breath model is built and before the method disclosed in aboveembodiment is implemented to register the ultrasound picture with themodality image. In detail, the breath model is implemented to correctdifferent frames of ultrasound images into the same breath depth (thesame breath state) so as to reach the purpose for reducing oreliminating the breath effect. The specific correcting process isdescribed below in detail.

If t is one frame of the selected frames selected by doctor forregistering, the breath depth corresponding to the frame is d(t),relative displacement of the target organ corresponding to the referencecoordinate system of the reference position is W(d(t)). If x is onepoint of the frame, located as R_(probe)(t)·A·x in the world coordinatesystem, in accordance with Formula (3), the position of the point istransformed as below when it is corrected to the reference breath depth:T(W(d(t)))·R _(probe)(t)·A·x  (8)

Wherein T(W(d(t))), which defined the breath depth d(t) corresponding tothe ultrasound image frame t as self-varying value, is the breathcorrection matrix corrected under the breath model. In this embodiment,T(W(d(t))) is acquired through the linear compensation of breath effectunder breath motion discipline in three dimensions for the relativedisplacement W(d(t)). If W(d(t))=(W_(x)(d(t))), W_(y)(d(t)),W_(z)(d(t))), the breath correction matrix under Homogeneous coordinatesis defined as:

$\begin{matrix}{T\left( {\left( {W\left( {d(t)} \right)} \right) = \begin{bmatrix}1 & 0 & 0 & {- {W_{x}\left( {d(t)} \right)}} \\0 & 1 & 0 & {- {W_{y}\left( {d(t)} \right)}} \\0 & 0 & 1 & {- {W_{z}\left( {d(t)} \right)}} \\0 & 0 & 0 & 1\end{bmatrix}} \right.} & (9)\end{matrix}$

In other embodiments, T(W(d(t))) could be acquired through compensationprocess after certain processes under the motion discipline in one ormore dimensions. Processes under the motion discipline could be selectedfrom non-linear transformation or giving different weights in differentdimensions.

In this embodiment, the breath model is implemented before theregistering process. The breath model is also able to be implementedafter the registering process in other embodiments, such as implementedin the fusion process. If the reference breath depth in registering isd₀, it is also capable of conducted real-time correction for breatheffect in the fusion process through above built breath model. Thecorrection algorithm is identical with the algorithm correctingdifferent frames into the same breath depth disclosed in aboveregistering process. The relationship between X_(us) and X_(sec) sec inthe fusion process conducted after correction is defined as Formula(10):X _(sec) =P·T(W(d(t)))·R _(probe)(t)·A·X _(us)  (10)

FIG. 8 is a schematic flow chart of registering and fusion processesafter breath module is built according to an embodiment of the presentdisclosure based on ultrasound images, position indicating informationof position sensor and position information of the breath sensor. Theflow chart shown in FIG. 8 comprises specific steps disclosed below:step S31 for sampling the ultrasound video data, step S32 for replayingthe ultrasound video data and selecting one or multiple image frames forregistering, step S33 for determining whether or not to conduct breathcorrection for the registered frames. If it is positive (YES), step S34is conducted for correcting the registered frames into the same breathdepth. If it is negative (NO), step S35 is conducted for registering theselected image frames. In step S36, it is determined that whether or notto conduct breath correction for the fusion result, if it is positive(YES), step S37 is conducted for correcting the fusion result into thesame breath depth and he step S38 is conducted thereafter. If it isnegative (NO), step S38 is conducted for displaying the fusion result.In the registering step, doctor is able to replay the video forselecting one or multiple ultrasound images for registering and acquirethe position indicating information of the position sensor forregistering. If the selected multiple ultrasound frames are underdifferent breath depth, it is able to correct the selected multipleultrasound frames into the same breath depth before the registeringstep. Alternatively, if doctor wants to observe the fusion result in thesame breath depth, it is able to correct it into the depth byimplementing the breath model.

In above embodiments for explaining how to correct the multipleultrasound frames into certain breath depth through the breath model andhow to correct the fusion result into certain breath depth, it isassumed that the target depth of the correction object is defined as thereference depth d₀ as the breath model is built. However, it isunderstandable that the correction depth of the target object could beany breath depth, and the breath correction matrix is transformed fromT(W(d(t))) to T⁻¹ (W(D))·T(W(d(t))).

A method for fusing at least one ultrasound image with a pre-storedmodality image disclosed in this embodiment comprises below steps: areplaying step for playing at least one ultrasound image frame from atleast one pre-stored ultrasound video data frame by frame under inputinstruction or replaying the ultrasound image frame above under inputinstruction, wherein the ultrasound video data comprises the ultrasoundimage frame sampled from the target object in at least one plane andposition indicating information corresponding to each of the ultrasoundimage frame, the position indicating information is generated by sensingthe position of a position sensor fixed with a ultrasound probe whileacquiring the ultrasound image, the breath position information isgenerated by sensing breath of the target object through the breathsensor fixed on the target object while acquiring the ultrasound image;

-   a registering step for registering the selected ultrasound image    frame with the modality image by implementing the position    indicating information corresponding to the selected ultrasound    image frame in the registering step, wherein the registering method    could be implemented by conventional image registering algorithm;    and-   an image fusing step for fusing the registered ultrasound image    frame with the modality image, wherein the multiple ultrasound    images are corrected into the same breath depth after registering    through the breath model so as to allow doctor to observe the breath    depth under the same breath depth.

The implement of how to build the breath model and how to correct imagesinto the same breath has been already disclosed in above embodiments.Therefore, corresponding descriptions related are omitted herein.

A method for fusing at least one ultrasound image with a pre-storedmodality image comprises below steps:

-   selecting multiple ultrasound image frames from at least one    ultrasound video data, defined as a selecting step, wherein the    ultrasound video data comprising the ultrasound image frame sampled    from a target object in at least one plane and position indicating    information corresponding to each of the ultrasound image frame, the    position indicating information is generated by sensing the position    of a position sensor fixed with a ultrasound probe while acquiring    the ultrasound image, the breath position information is generated    by sensing breath of the target object through the breath sensor    fixed on the target object while acquiring the ultrasound image;-   building a breath model according to the ultrasound video data,    defined as a breath model-building step;-   registering the selected ultrasound image frame with a modality    image, defined as a registering step; and-   image fusing the registered ultrasound image frame with the modality    image, defied as a image fusing step, wherein the breath-correcting    step is conducted in the condition selected from conducting the    breath-correcting step before the registering step or conducting the    breath-correcting step after the image fusing step.

In one embodiment, a step for building the breath model comprises belowsteps:

-   selecting a ultrasound image frame corresponding to a reference    breath depth as a reference frame for each portion of the ultrasound    video data to acquire a motion amount of the target object    corresponding to other ultrasound frames with respect to the    reference frame, and calculating a relative motion amount of the    target object corresponding to other ultrasound frames in a    reference coordinate system under different breath depths, which    defined as a relative motion-calculating sub-step; and-   fitting the different breath depth with the relative motion amount    in the reference coordinate system under different breath depths to    build the breath model, defined as a fitting sub-step.

In another embodiment, the step of correcting multiple ultrasound framesinto the same breath depth further comprises:

-   for any ultrasound image frame of the multiple ultrasound video,    acquiring the breath depth corresponding to the ultrasound frame by    implementing the corresponding breath position information and the    relative motion amount of the target object in the ultrasound frame    under the reference coordinate system of the reference position    according to the breath model, which defined as a calculating    sub-step; and-   correcting the relative motion amount in the coordinate system    corresponding to the multiple ultrasound frames to a predetermined    breath depth according to the breath model, defined as a correcting    sub-step.

Detail descriptions of above steps and sub-steps could be referred tocorresponding parts in above embodiments and are be omitted herein. Themultiple ultrasound image frames processed in the registering step inthis embodiment could be selected from the multiple image framesselected when the breath model is built or the multiple image framesacquired based on the real-time ultrasound images.

As described above, for resolving two disadvantages of conventionalregistration implementations, that are low registering accuracy andfusing success rate based on real-time ultrasounds and poor performancefor eliminating breath effect, a registering and fusing methodimplemented by replaying ultrasound video data with the position sensorinformation is provided at the present application. A breath model isbuilt for breath correction in the mean time. For conducting breathcorrection, patient should be asked to breath normally and theultrasound videos for each of the position overlapping with includedangles among each other for more than one breath period is sampled torecord sensor position information of the sensor on the probe,indicating information corresponding for each of the ultrasound imageframes and breath sensor position information for the breath sensorfixed on patient's abdomen. The patient's breath model is builtaccording to the video and the sensor information hereafter.

Before the registering step is conducted, in the sampled ultrasoundvideo data containing information for registering, each of theultrasound images contains information of the sensor positioninformation and the indicating information of the sensor fixed with theultrasound probe and the breath sensor position information for thebreath sensor fixed on patient's abdomen. In the registering step,doctor is allowed to replaying the video for searching the videocontinuously or frame by frame so as to select one or more image framesfor registering, and the corresponding position information of theposition sensor fixed with the ultrasound probe is acquired forregistering by the navigation system at the same time. When the multipleframes are selected for registering, doctor is allowed to correct themultiple frames into the same breath state by implemented the breathemodel and the breath sensor information. In the fusion step, the resultis corrected in real-time according to the breath sensor information byimplemented the breath model so as to reach the purpose of eliminatingor reducing the breath effect.

The image fusion method disclosed in the present application registersand fuses the ultrasound video with other modality data and corrects thefusion result with the modality data for breath correction. The imagefusion method disclosed in the present application is not only able tobe implemented in examination for liver but also able to be implementedin examination for other abnormal organs such as kidney or prostate.

It is understandable for the skilled in the art that all or some of theprocesses disclosed in the embodiments of the present application areable to be implemented by instructing relating hardware through computerprograms. Above programs are able to be stored in a readable storingmedia of computer. Above programs are able to include the implement ofall flow charts for all methods disclosed in above embodiments inexecution. The readable storing media include but not limited to selectfrom below: Hard Disc, Optical Disc, Read-Only Memory (ROM) and RandomAccess Memory (RAM).

Although the present disclosure has been described through specificembodiments, the present disclosure is not limited to the specificembodiments described above. Those of skill in the art should understandthat various modifications, alternatives and variations may be madebased on the present disclosure, which all should be within the scope ofprotection of the present disclosure. Furthermore, “a (an) embodiment”or “another embodiment” mentioned above may represent differentembodiments, or may also be combined completely or partly in oneembodiment.

The invention claimed is:
 1. A method for fusing at least one ultrasoundimage with a pre-stored modality image comprising: selecting multipleultrasound image frames from at least one portion of pre-storedultrasound video data, defined as a selecting step, wherein theultrasound video data comprises ultrasound image frames sampled from atarget object along at least one plane and a position indicatinginformation corresponding to each of the ultrasound image frames at asampling period, wherein the position indicating information isgenerated by sensing a position of a position sensor fixed with anultrasound probe while sampling the ultrasound image frames; registeringthe at least one selected ultrasound image frame with the modalityimage, defined as a registering step, by implementing the positionindicating information corresponding to the at least one selectedultrasound image frame in the registering step to create a registeredultrasound image; and fusing the registered ultrasound image frame withthe modality image, defined as an image fusion step; building a breathmodel according to the ultrasound video data, defined as a breath modelbuilding step, using the position indicating information correspondingto each of the ultrasound image frames, wherein in each of theultrasound image frames, a breath depth is defined as a movement of abreath sensor relative to a reference position of the breath sensor, thebreath sensor is configured to be fixed on the target object; andcorrecting, using the breath model, the multiple ultrasound image framesto have a same breath depth across the multiple ultrasound image framesin a breath-correcting step occurring at either or both before theregistering step and during the fusing step, wherein the ultrasoundvideo data further comprises breath position information correspondingto each of the ultrasound image frames, the breath position informationis generated by sensing the breath of the target object through thebreath sensor while sampling the ultrasound image frames; and whereinthe breath model-building step comprises: selecting an ultrasound imageframe corresponding to a reference breath depth at the referenceposition on a motion path of the breath sensor fixed on the targetobject as a reference frame for each portion of the ultrasound videodata and acquiring a motion amount of the target object corresponding toother ultrasound image frames with respect to the reference frame, andcalculating a relative motion amount of the target object correspondingto other ultrasound image frames in a same reference coordinate systemunder different breath depths, based on the motion amount, defined as arelative motion-calculating sub-step; and fitting the different breathdepths and relative motion amounts, including the relative motionamount, in the reference coordinate system under the different breathdepths to build the breath model, defined as a fitting sub-step.
 2. Themethod of claim 1, wherein the relative motion amount of the targetobject corresponding to other ultrasound image frames in the referencecoordinate system under different breath depths satisfies: an errorbetween the motion amount of the target object corresponding to thereference frame and the relative motion amount under referencecoordinate system at a predetermined breath depth is minimized.
 3. Themethod of claim 2, wherein the motion amount is a rigid motion amountdefined as part of a periodic motion, the relative motion amount underthe reference coordinate system is an amount of displacement, therelative motion amount of the target object corresponding to otherultrasound image frames in the reference coordinate system underdifferent breath depths is calculated according to the followingformula:$\underset{W{(D)}}{\arg\;\min}\left( {\sum\limits_{i = 1}^{n}\left( {\sum\limits_{{{All}\mspace{11mu}{d{(t)}}} = D}\left( {f\left( {{{proj}_{i}\left( {W(D)} \right)} - {V_{i}\left( {d(t)} \right)}} \right)} \right)} \right)} \right)$wherein the breath depth is d(t), V_(i)(d(t)) is a displacement amountof the target object relating to the reference frame, W(D) is adisplacement amount of the target object under the breath depth D,proj_(i)(W(D)) is a projection component of W(D) in a planecorresponding to the reference frame, n is the number of all portions ofthe ultrasound video data, i=1,. . . ,n, ƒ(proj_(i)(W(D))−V_(i)(d(t)) isan error function of proj_(i)(W(D)) and V_(i)(d(t))).
 4. The method ofclaim 1, wherein the step of correcting multiple ultrasound frames intothe same breath depth comprises: for any ultrasound frame of themultiple ultrasound frames, acquiring the breath depth corresponding tothe ultrasound frame by implementing the corresponding breath positioninformation, and acquiring the relative motion amount of the targetobject in the ultrasound frame with respect to the reference positionaccording to the breath model, which defined as a calculating sub-step;and correcting, in a breath correction step, the relative motion amountin the coordinate system corresponding to the multiple ultrasound framesto a predetermined breath depth according to the breath model, which isdefined as a correcting sub-step.
 5. The method of claim 4, wherein themotion amount is a rigid motion amount defined as part of a periodicmotion, the relative motion amount in the reference coordinate system isa displacement amount, and the correcting sub-step comprises: acquiringtranslation vectors of the target object in each dimension of thereference coordinate system according to the displacement amount;determining a rotating factor according to the breath model; acquiring abreath correction matrix according to the translation vectors and therotating factor; and correcting the relative motion amount in thereference coordinate system corresponding to the multiple ultrasoundframes to a predetermined breath depth according to the breathcorrection matrix.
 6. The method of claim 4, wherein the step ofcorrecting the relative motion amount in the reference coordinate systemcorresponding to the multiple ultrasound frames to the predeterminedbreath depth comprises: for any point on any frame of the ultrasoundimage frames, calculating a position of the point in the referencecoordinate system and calculating a position of the point in thepredetermined breath depth using P·T(R_(probe)·A·X_(us)), wherein P is atransformation matrix from a ultrasound image space to a modality imagespace, T is a space mapping matrix for correction, R_(probe) is atransformation matrix from a position sensor space to a world coordinatesystem, A is a transformation matrix from the ultrasound image space tothe position sensor space, and X_(us) is a coordinate of the point inthe ultrasound image space.
 7. The method of claim 5, wherein the imagefusion step after the breath correction step is calculated according to:X _(sec) =P·T(W(d(t)))·R _(probe)(t)·A·X _(us) wherein X_(us) is acoordinate of the point in the ultrasound image space, X_(sec) is acoordinate of the point in the modality image, P is a transformationmatrix from a ultrasound image space to a modality image space,T(W(d(t))) is a breath correction matrix, R_(probe)(t) is atransformation matrix from a position sensor space to a world coordinatesystem, A is a transformation matrix from the ultrasound image space tothe position sensor space.
 8. The method of claim 1, further comprising:displaying multiple registered ultrasound image frames or fusedultrasound image frames, and at least one intersection line and includedangle among the multiple registered or fused ultrasound image frames atthe same time.
 9. An ultrasound image fusion navigation system,comprising: a probe and a position sensor fixed with the probe; a breathsensor configured to be fixed on a target object; a sampling unit whichsamples the target object along at least one plane at a sampling periodto acquire at least one portion of pre-stored ultrasound video datacomprising information for registering, and recording a positionindicating information for each of ultrasound image frames of thepre-stored ultrasound video data, wherein the position indicatinginformation is generated by sensing a position of the position sensorwhile sampling the ultrasound image frames; a selecting unit whichselects multiple ultrasound image frames from the pre-stored ultrasoundvideo data according to an input instruction; a registering unit whichregisters the selected multiple ultrasound image frames with a modalityimage by implementing the position indicating information of the atleast one ultrasound image frame; a fusion unit which fuses theregistered ultrasound image frame with the modality image; a breathmodel building unit which builds a breath model according to theultrasound video data, using the position indicating informationcorresponding to each of the ultrasound image frames, wherein in each ofthe ultrasound image frames, a breath depth is defined as a movement ofa breath sensor relative to a reference position of the breath sensor; acorrecting unit which corrects, using the breath model, the multipleultrasound image frames to have a same breath depth across the multipleultrasound image frames; wherein the sampling unit further recordsbreath position information corresponding to each of the ultrasoundimage frames, the breath position information is generated by sensingthe breath of the target object through the breath sensor while samplingthe ultrasound image frames; and wherein the breath model building unitbuilds the breath model by: selecting an ultrasound image framecorresponding to a reference breath depth at the reference position on amotion path of the breath sensor fixed on the target object as areference frame for each portion of the ultrasound video data andacquiring a motion amount of the target object corresponding to otherultrasound image frames with respect to the reference frame, andcalculating a relative motion amount of the target object correspondingto other ultrasound image frames in a same reference coordinate systemunder different breath depths, based on the motion amount; and fittingthe different breath depths and relative motion amounts, including therelative motion amount, in the reference coordinate system under thedifferent breath depths to build the breath model.